Video browser based on character relation

ABSTRACT

A video data structure and a video browser for video browsing are disclosed. The data structure includes segment group information indicating that a first segment group represents highlights sharing at least two objects in a program&#39;s content. The segment group information is used to display the highlights represented by the first segment group. The present invention can be implemented to allow users to easily understand relations between characters and changes in relations between characters in a movie or a drama by displaying significant events in the corresponding relations and by displaying character relations in a tree structure.

CROSS REFERENCE TO RELATED APPLICATIONS

The present application is a continuation of co-pending application Ser.No. 09/645,613, filed Aug. 25, 2000, the entirety of which isincorporated by reference herein. This application claims the benefit ofKorea Patent Application No. 35688/1999, filed Aug. 26, 1999.

BACKGROUND

1. Field of the Invention

The present invention relates to a video browser, and more particularlyto a video browser for browsing a video based on relations betweencharacters.

2. Background of the Related Art

Typically, users simply view movies and/or dramas as broadcasted througha TV or played at a movie theater. However, a user may wish to view aparticular movie or drama at a particular time, or wish to view only aparticular section of a movie or a drama. Accordingly, varioustechniques which enable selective watching of a movie/drama or sectionsof a movie/drama have been suggested.

In the related art, for example, various video data may be representedor classified into format portion, index portion, media portion, segmentportion, target portion, and/or representation portion. Also, data onvarious characters or objects, such as a name of an object, position onthe screen, or numeric data with relation to a segment of the video datain which the object appears, may be represented by the target andrepresentation chunk. Accordingly, a user can select an object through atable and reproduce for display a particular segment where the object isshown in the video.

In another related art, various additional data of a video data areobtained before, during or after the production of the video data.Thereafter, an additional information table of the obtained data iscomposed and provided to users. Namely, the additional data table mayinclude a position where an actor appears, a position where a characterof the actor appears, and a position where stage properties appear, suchthat a scene can be reproduced as selected by a user through theadditional data table. For example, if a user selects a stage property,information on the selected stage property, such as the manufacturer andprice, may be displayed on a screen, and the user may be able to connectwith the manufacturer or a seller of the stage property through anetwork connection.

In still another related art, recording information on each segment of avideo in a video map has been suggested. That is, information, such asthe degree of violence, the degree of adult content, the degree ofimportance of content, characters positions, and the degree ofdifficulty in understanding, may be indicated for each segment of avideo in the video map. Thus, the user may set a degree of preferencefor one or more items of the video map, and only segments of the videomeeting the set degree of preference would be reproduced, therebylimiting a display of particular contents to unauthorized viewers.

Similarly, other techniques in the related art as described aboveprovide items simply arranged without any relation to the objectsappearing in the movie or drama, based upon the selection of the user.However, the content of a movie or drama generally builds aroundrelations between characters, places and events. For example, relationsbetween characters may not change from beginning to the end of the storyor may continuously vary. Moreover, since one or more characters relateto a specific character in the movie or drama, the browsing method inthe related art substantially fails to provide an accurate understandingof the story of the movie or drama to the user.

Therefore, techniques in the related arts have disadvantages in that itis difficult to understand a video centering on relations amongcharacters according to the development of events, changes of relations,and relations among characters and places as events develop.

SUMMARY

Accordingly, an object of the present invention is to solve at least theproblems and disadvantages of the related art.

Another object of the present invention is to provide a video browser inwhich event segments showing changes in relations between characters aresummarized and displayed based on a video data structure which includescharacters in a video that significantly act on development of an event,and includes a constant and variable relations between characters.

A still another object of the present invention is to provide a videobrowser in which main events that significantly act on variablerelations between characters are connected with corresponding variablerelations based on a video data structure which includes characters in avideo that significantly act on development of an event, and includes aconstant and variable relations between characters.

A further object of the present invention is to provide a video browserin which a key frame of event segments which show a correspondingrelation between characters is displayed based on a video data structurewhich includes characters in a video that significantly act ondevelopment of an event, and includes a constant and variable relationsbetween characters.

A further object of the present invention is to provide a video browserin which a constant relation and changes in relations between charactersare displayed depending on preset relation types based on a video datastructure which includes characters in a video that significantly act ondevelopment of an event, and includes a constant and variable relationsbetween characters.

Additional advantages, objects, and features of the invention will beset forth in part in the description which follows and in part willbecome apparent to those having ordinary skill in the art uponexamination of the following or may be learned from practice of theinvention. The objects and advantages of the invention may be realizedand attained as particularly pointed out in the appended claims.

To achieve the objects and in accordance with the purposes of theinvention, as embodied and broadly described herein, a video browsingsystem for browsing a video based on a data structure in which aconstant and variable relations between characters of a video areconnected with characters and corresponding event segments, comprises(a) browsing and displaying a video segment showing a constant relationbetween characters and variable relations between characters based on anevent segment showing a corresponding constant relation and variablerelation, and (b) displaying a video segment of a corresponding eventsegment in response to a user's selection.

The video browser of the present invention is characterized in thatevent segments showing a constant relation between characters andchanges in the relations is summarized and displayed. Also, main eventsthat significantly act on variable relations are connected withcorresponding variable relations and characters.

Moreover, the video browser of the present invention is characterized inthat constant relation and variable relation between the characters arebrowsed and displayed as a constant relation and a variable relationbetween characters selected by the user. The constant relation andvariable relation between characters may also be browsed and displayedas a constant relation and a variable relation between every charactersrelated with a selected character.

In still another video browser of the present invention, the constantrelation and variable relation between characters are displayed in atree structure. The video browser of the present invention may display aconstant relation and variable relation between characters depending onpreset relation types. The relation types may be set in semantic typessuch as “family relation,” “business relation,” and “social relation.”

Furthermore, the video browser of the present invention is characterizedin that event segments showing constant and variable relations betweencharacters are displayed as key frames. Finally, each key frame may showa corresponding relation between characters.

BRIEF DESCRIPTION OF THE DRAWINGS

The present invention will be described in detail with reference to thefollowing drawings in which like reference numerals refer to likeelements wherein:

FIG. 1 shows an example of a video data structure based on characterrelation according to the present invention;

FIG. 2 shows an example of a video data structure in which eventsshowing character relations are summarized in accordance to the presentinvention;

FIG. 3 shows a video browser based on the video data structure of FIG.2;

FIG. 4 shows a video data structure of an event segment showing variablerelations between characters;

FIG. 5 shows a video browser based on the data structure of FIG. 4 inaccordance with the present invention;

FIG. 6 shows an example of a video browser according to the presentinvention; and

FIG. 7 shows another example of a video browser according to the presentinvention.

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS

Reference will now be made in detail to the preferred embodiments of thepresent invention, examples of which are illustrated in the accompanyingdrawings.

FIG. 1 shows a video data structure based on character relations inaccordance with the present invention for a video browser based oncontent. A video browser based on content is disclosed in U.S. Pat. No.6,602,297, entitled “Motional Video Drowsing Data Structure and BrowsingMethod Therefor,” and is fully incorporated herein.

Referring to FIG. 1, a visual description scheme (DS) 101 is dividedinto a visualization DS 102, a syntactic structure DS 103, and asemantic structure DS 104. The visualization DS 102 is organized into ahighlight view DS 105 for displaying a summarized data as a videohighlight, and a key frame view DS 106 for displaying summarized databased on a key frame. Because, a video plot can be summarized briefly orwith greater amounts of detail, the highlight view DS 105 enables adisplay of multi-levels of summarized data with a level 107, a view type108 and a time DS 109. Similarly, the key frame view DS 106 enables adisplay of multi-levels of summarized data with a level 110 and a timeDS 111.

Namely, the highlight view DS 105 is organized into a level 107 whichhas multiple levels of highlight data based, upon a degree of detail insummarizing a video. In such case, summary data in each level mayinclude a corresponding period in the time DS 109 for use in highlight.The key frame view DS 106 is also organized into a level 110 which hasmultiple levels of summarized data based upon a degree of detail. Insuch case, summary data in each level may include a corresponding periodin the time DS 111 for use as a key frame.

The syntactic structure DS 103 is used for displaying the actual videoand includes actual video segments to be displayed. The syntacticstructure DS 103 is organized into actual video segments in segment DSs112, and corresponding temporal positions of video segments in the timeDS 113.

The semantic structure DS 104 includes additional information describinga video, and is organized into an event DS 114 including eventinformation, an object DS 117 including object information, and anevent/object relation graph DS 119 including information describingrelations between objects and places, and corresponding events to therelations. Namely, the event DS 114 describes events, and the object DS117 describes objects such as characters and places. The event/objectrelation graph DS 119 describes a constant relation or changes in arelation between characters, a relation between an object and a place,or a relation between an object and an event.

Here, a constant relation means either a relation between charactersthat cannot change throughout a video, such as a parent to childrelation, or a relation which is most representative of the relationsbetween characters.

When an event is selected to display a video segment corresponding tothe event, the event DS 114 of the semantic structure DS 104 is used.The event DS 114 is divided into a Reference to Segment 115 includinginformation necessary for displaying a segment of a video correspondingto the selected event and an annotation DS 116 including informationwhich connects events with actual positions of the events in a video andinformation for explaining events in a video. Particularly, the event DS114 is used in displaying a selected event and connects the selectedevent with a position of the event in a video data through the Referenceto Segment 115, and the contents of a selected event is annotatedthrough the annotation DS 116.

The object DS 117 is organized into an object annotation DS 118including information for describing objects such as characters orplaces. That is, the object DS 117 is used in displaying characters, anddetailed information of the characters can be summarized or annotatedthrough the annotation DS 118.

The event/object relation graph DS 119 includes information on aconstant relation and variable relation between characters. Theevent/object relation graph DS 119 is organized into an entity relation120 with a return which allows a display of character relations in atree structure in which a relation may have subordinate relations.According to the present invention, constant relations betweencharacters are placed in an upper level of the tree while changes ofrelations between the characters are placed in a lower level of thetree.

The entity relation 120 is also divided into a relation 121, a Referenceto Object 124, and a Reference to Event 125. The relation 121 isorganized into a name 122 including information on the titles ofrelations, and a type 123 including information on the nature ofrelations. For example, a nature of relation may be ‘family’ and a titleof relation may be ‘spouse.’ The Reference to Object 124 connectsrelated characters with each other and the Reference to Event 125connects events which shows particular relations.

In the above video data structure, the notation above each data such as{0,1}, {0,*}, or {1,*} indicates the number of data for thecorresponding data. For example, the notation of {0,1} for thevisualization DS 102 indicates that the visual DS 101 can have zero orone visualization DS. On the other hand, the notation of {0,*} for thesegment DS 112 indicates that the syntactic structure DS 103 may havefrom zero to any number of segment DS.

FIG. 2 shows a method of using a data structure of FIG. 1 to easilyunderstand and browse a video based on relations between characters.Referring to FIG. 2, a representative relation (constant relation)between ‘character 1’ and ‘character 2’ is placed at the top of the treestructure, and variable relations (relation 1˜relation 4) between‘character 1’ and ‘character 2’ are placed at the bottom of the treestructure. Also, main events which are significant in a change ofrelation in a variable relation are placed between variable relations.The main events are summarized video data and are displayed by highlightdata.

Here, the events are connected through the Reference to Event 125 in theentity relation 120 and the video segments are connected through theReference to Segment 115 in the event DS 114. Thereafter, an actualvideo segment Is displayed by connecting the time DS 113 of the segmentDS 112 with the time DS 109 of the level 107 in the highlight DS 105 ofthe visual DS 102.

FIG. 3 shows an example screen of a video browser based on the videodata structure of FIG. 2. Referring to FIG. 3, ‘character n’ of a videois displayed on a character screen 301, and a video is displayed on amain screen 302 according to a selection received through a userinterface 303.

For example, if a user selects ‘character 1’ and ‘character 3’ from thecharacter screen 301, main events which are significant in bringingabout a change in the variable relations between ‘character 1’ and‘character 3’ can be browsed based on the data structures of FIGS. 1 and2. As a result, contents corresponding to the main events are summarizedand displayed on the main screen 302.

FIG. 4 shows another method of using a video data structure includinginformation on variable relations of characters and main eventssignificant in the variable relations.

Referring to FIG. 4, a representative relation (constant relation)between ‘character 1’ and ‘character 2’ is placed at a top of a treestructure, and variable relations (relation 1˜relation 4) between‘character 1’ and ‘character 2’ are placed at the bottom of the treestructure. Main events (event 1˜event 3) which are significant inbringing about a change in the variable relations are placed between thevariable relations. The method of FIG. 4 is realized in such a mannerthat a main event segment which brings a new relation is connected withthe variable relations through the Reference to Event 125 of the entityrelation 120.

FIG. 5 shows a video browser in accordance with the present inventionbased on the data structure of FIG. 4. Referring to FIG. 5, charactersin a video are displayed on a character screen 501, and connectionbetween variable relations and events are displayed on a characterrelation-variable event screen 502. Also, an event segment correspondingto a variable relation can be reproduced and displayed on a main screen503.

For example, if a viewer selects ‘character 1’ and ‘character 3,’variable relations between ‘character 1’ and ‘character 3,’ and mainevents which are significant in bringing about a change in the variablerelation are respectively displayed on the screen 502. At this time, arelation or event may be displayed by an key frame or annotation. Also,a video segment corresponding to an event such as ‘event 2’ selectedfrom the screen 502 is reproduced and displayed on the main screen 503.

FIG. 6 shows another example screen of a video browser according to thepresent invention. Referring to FIG. 6, main characters of a video aredisplayed on a character screen 602. Also, characters having relationswith a character selected from the character screen 601, and a constantrelation and variable relations between the selected character andrelated characters are displayed on a relation screen 602.

At this time, a constant relation between the selected character and arelated character is displayed on the top of a relation tree structurewhile variable relations between the selected character and the relatedcharacter is displayed on the bottom of the tree structure.

Furthermore, a key frame of events significant in both the constantrelation and variable relations is displayed on a main scene screen 603.Here, key frames of event segment corresponding to a type(s) of eventmay be displayed on the main scene screen according to a user selectionthrough a selection screen 604. Namely, key frames of main eventsegments which directly show a selected relation or key frames ofsecondary event segments which indirectly show the selected relation, orboth can be displayed according to the user selection. For example, if aselected relation is a husband and wife, a main event segment may be avideo segment of a marriage ceremony while a secondary event segment maybe a video segment of third parties discussing the marriage ceremony. Anevent segment corresponding to an event, for example a key frame,selected from the main scene screen 603 can thus be reproduced anddisplayed on the main screen 605.

For example, when a user selects ‘character 1’ from among the charactersin the character screen 601, other characters ‘character 2’˜‘character4’ related with ‘character 1’ are displayed on the relation screen 602.If ‘relation 2’ with ‘character 2’ is selected from the relation screen602, event segments corresponding to ‘relation 2’ with ‘character 2’ isdisplayed on the main scene screen 603 as key frames. Also, a period ofa video corresponding to ‘event 6’ selected from the main scene screen603 is reproduced and displayed on the main screen 605.

At this time, the video browser of FIG. 6 can be implemented by settinga relation type in the data structure of FIG. 1. In other words, eventsconnected with a relation may vary depending on whether the type 123 ofthe relation 121 is the direct or indirect type as discussed above.Also, the events are connected through the Reference to Event 125 of theentity relation 120 and event segments are connected through theReference to Segment 115 of the event DS 114. Thereafter, an actualvideo segment is displayed by connecting the time DS 113 of the segmentDS 112 with the time DS 111 of the level 110 in the key frame view DS106 of the visual DS 102.

FIG. 7 shows still another example screen of a video browser accordingto the present invention. Referring to FIG. 7, characters of a video aredisplayed on a character screen 701; and character having relations witha selected character, and a constant relation and a variable relationbetween the selected character and related characters are displayed on arelation screen 702. Here, the constant and variable relations maycategorized into different natures of relations and relationscorresponding to a particular nature(s) of relation selected through aselection screen 703 may be displayed in the relation screen. The natureof relation may be a family relation, a business relation, or a socialrelation.

Thus, a key frame of events significant in the selected nature(s) ofrelation is displayed on a main scene screen 704. A video segmentcorresponding to the selected relation and event is reproduced anddisplayed, on a main screen 705, For example, when a user selects‘character 1’ from characters displayed on the character screen 701,characters (‘character 2’˜‘character 4’) ‘aving a social relation with“character 1” may be displayed and an event segment of ‘event 6’corresponding to ‘social relation 2’ with ‘character 2’ can bereproduced and displayed on the main screen 705.

As in FIG. 6, a constant relation, between a selected character and arelated character is displayed on the top of a relation tree structurewhile variable relations between the selected character and the relatedcharacter is displayed on the bottom of the tree structure.

The video browser of FIG. 7 can be realized by setting a relation type123 of the relation data 121 in the data structure of FIG. 1. In otherwords, events related with the relation data 121 may vary depending onwhether the relation type 123 of the relation data 121 is a ‘familyrelation,’ a ‘business relation,’ or a ‘social relation.’ As a result,browsing of FIG. 7 can be performed. At this time, the events aremutually connected through the reference to event 125 of the realrelation 120. Segments are mutually connected through the reference tosegment 115 of the event DS 114. Then, the time DS 113 of the segment DS112 is connected with the time DS 111 of the level 107 displayed in thekey frame DS 106 of the visual DS 102.

As aforementioned, the video browser based on a character relationaccording to the present invention has the following advantages. Theevent periods of the variable relation between the characters aresummarized and displayed, and the main events concerned in the variablerelation between the characters are connected between the variablerelations and then displayed.

Furthermore, the data structure for browsing a video is based on arelation between characters and a variable relation between them. A keyframe of an event period showing a corresponding relation between thecharacters clearly or directly, or a key frame of an event periodshowing it with hint or indirectly can be displayed based on the datastructure. Also, the relations between the characters are divided into afamily relation, a social relation, and an emotion relation to beselectively provided to a user, so that video browsing can be performedbased on the relations between the characters.

Accordingly, contents of the video can be browsed based on charactersthat significantly act on development of a story of a movie or drama,the relation between the characters, and the variable relation betweenthem. After all, the user can easily browse the video based oncharacters, event and relation which are concerned in development of astory and important for contents of a real video. Also, the user canunderstand and browse a desired video in various types based on thevideo data structure which represents characters and their relation.

Furthermore, the present invention may be applied to a VOD system in thebroadcasting field so that the user can view a desired part. This couldlead to an advantage that the user can reproduce and view the desiredpart within short time and to effective utilization of the networksource. Also, the present invention may be applied to a video player forhome use and a broadcasting video player so that the video browsingenvironment can be provided to easily browse a desired part of a movieor drama stored in the video player.

The foregoing embodiments are merely exemplary and are not to beconstrued as limiting the present invention. The present teachings canbe readily applied to other types of apparatuses. The description of thepresent invention is intended to be illustrative, and not to limit thescope of the claims. Many alternatives, modifications, and variationswill be apparent to those skilled in the art.

1. A method for processing multimedia data, the method comprising:receiving a content description for a multimedia program, the contentdescription including segment group information identifying a firstsegment group that includes a plurality of segments from the multimediaprogram, the segment group information indicating that the first segmentgroup represents highlights sharing at least two objects in the contentof the multimedia program; and using the segment group information todisplay the highlights represented by the first segment group.
 2. Themethod of claim 1, wherein the first segment group represents highlightssharing at least two characters in the content of the multimediaprogram.
 3. The method of claim 1, wherein the content descriptionspecifies a duration for a running time of the highlights represented bythe first segment group.
 4. The method of claim 1, wherein the segmentgroup information identifies a second segment group as an alternativegroup that represents highlights from the content of the multimediaprogram.
 5. The method of claim 4, wherein the segment group informationspecifies different durations for the first and second segment groups.6. The method of claim 4, wherein the first and second groups representhighlights with different levels of detail.
 7. The method of claim 1,wherein the content description includes segment information.
 8. Themethod of claim 7, wherein the segment information specifies arespective starting point for each segment in the first segment group.9. A system for processing multimedia data, the system comprising: adisplay device; and data processing apparatus configured to performoperations comprising: receiving a content description for a multimediaprogram, the content description including segment group informationidentifying a first segment group that includes a plurality of segmentsfrom the multimedia program, the segment group information indicatingthat the first segment group represents highlights sharing at least twoobjects in the content of the multimedia program; and using the segmentgroup information to display the highlights represented by the firstsegment group on the display device.
 10. The system of claim 9, whereinthe first segment group represents highlights sharing at least twocharacters in the content of the multimedia program.
 11. The system ofclaim 9, wherein the content description specifies a duration for arunning time of the highlights represented by the first segment group.12. The system of claim 9, wherein the segment group informationidentifies a second segment group as an alternative group thatrepresents highlights from the content of the multimedia program. 13.The system of claim 12, wherein the segment group information specifiesdifferent durations for the highlights represented by the first andsecond segment groups.
 14. The system of claim 12, wherein the first andsecond groups represent highlights with different levels of detail. 15.The system of claim 9, wherein the content description includes segmentinformation.
 16. The system of claim 15, wherein the segment informationspecifies a respective starting point for each segment in the firstsegment group.
 17. A computer program product for processing multimediadata, the computer program comprising instructions to cause dataprocessing apparatus to perform operations comprising: receiving acontent description for a multimedia program, the content descriptionincluding segment group information identifying a first segment groupthat includes a plurality of segments from the multimedia program, thesegment group information indicating that the first segment grouprepresents highlights sharing at least two objects in the content of themultimedia program; and using the segment group information to displaythe highlights represented by the first segment group.
 18. The computerprogram product of claim 17, wherein the first segment group representshighlights sharing at least two characters in the content of themultimedia program.
 19. The computer program product of claim 17,wherein the content description specifies a duration for a running timeof the highlights represented by the first segment group.
 20. Thecomputer program product of claim 17, wherein the segment groupinformation identifies a second segment group as an alternative groupthat represents highlights from the content of the multimedia program.21. The computer program product of claim 20, wherein the segment groupinformation specifies different durations for the first and secondsegment groups.
 22. The computer program product of claim 20, whereinthe first and second groups represent highlights with different levelsof detail.
 23. The computer program product of claim 17, wherein thecontent description includes segment information.
 24. The computerprogram product of claim 23, wherein the segment information specifies arespective starting point for each segment in the first segment group.